Optional: Attention Mechanisms

Back to Home

01. Introduction to Attention
02. Encoders and Decoders
03. Elective: Text Sentiment Analysis
04. Sequence to Sequence Recap
05. Encoding -- Attention Overview
06. Decoding -- Attention Overview
07. Attention Overview
08. Attention Encoder
09. Attention Decoder
10. Attention Encoder & Decoder
11. Bahdanau and Luong Attention
12. Multiplicative Attention
13. Additive Attention
14. Additive and Multiplicative Attention
15. Computer Vision Applications
16. Other Attention Methods
17. The Transformer and Self-Attention
18. Notebook: Attention Basics
19. [SOLUTION]: Attention Basics
20. Outro

Back to Home

10. Attention Encoder & Decoder

In machine translation applications, the encoder and decoder are typically

Generative Adversarial Networks (GANs)

Recurrent Neural Networks (Typically vanilla RNN, LSTM, or GRU)

Mentats

SOLUTION:

Recurrent Neural Networks (Typically vanilla RNN, LSTM, or GRU)

Word Embeddings

What's a more reasonable embedding size for a real-world application?

4

200

6,000

SOLUTION:

200

What are the steps that require calculating an attention vector in a seq2seq model with attention?

Every time step in the model (both encoder and decoder)

Every time step in the encoder only

Every time step in the decoder only

SOLUTION:

Every time step in the decoder only

udacimak v1.3.0